Make stored day-records timezone-independent with CalendarDay#85
Conversation
Introduce a timezone-independent calendar day (year/month/day) as the stable identity of a logical day, with Date conversions, ISO string round-tripping, pure-Gregorian arithmetic/ranges, and a legacy start-of-day recovery init for the upcoming data migration. Plan step: Add CalendarDay + tests.
Make the timezone-independent CalendarDay the identity of a logical day throughout the Where domain, so user-asserted records and dismissals stop drifting onto a different day (and reappearing as data errors) when the device changes time zones. - DayPresence is now keyed by `day: CalendarDay` (with `startOfDay(in:)` and a `date:in:` producer convenience); Codable decodes legacy `date` manifests via UTC recovery. - Aggregation, YearReport, RegionDayLocations, MissingDays/MissingDayRange, PresenceCalendar, and the data-issue detectors/IDs/keys all work in CalendarDay. DataIssueID.storageKey is now `prefix:YYYY-MM-DD`, stable across travel. EvidenceReader day keys are CalendarDay. - Persistence: SDManualDay gains canonical `dayKey` (ISO string); `dateKey` is retained as informational + the migration source. Store CRUD keys manual days by CalendarDay (lexicographic ISO range for year scoping); WhereStore/DayJournal/ReportReader/WidgetDataReader updated to match. - WhereUI/WhereIntents consumers project CalendarDay -> Date only for display and DatePicker bindings. Adds a reusable boot-time migration framework (StoreMigration registry + App-Group version marker + SwiftDataStore.runPendingMigrations, hooked in WhereBootstrap.makeServices) and the v1 CalendarDayMigration: backfills dayKey from legacy instants via a +12h noon-nudge recovery (robust to the writer's zone) and rewrites legacy epoch dismissal keys to ISO form. Plan steps: DayPresence/aggregation; MissingDays + calendar; detection keys; persistence; migration framework; CalendarDayMigration; WhereUI consumers.
- WhereCore/AGENTS.md: invariants for CalendarDay (logical-day identity vs Date-for-instants) and the boot-time StoreMigration pattern. - Where/AGENTS.md: CalendarDay note in Dates & presentation. - WhereCore/README.md: CalendarDay + StoreMigration/CalendarDayMigration in the API tour; dismissals key on a TZ-independent storageKey. - Add TimeZoneIndependenceTests: a backfilled day read in a different time zone stays present and is not re-flagged as missing; dismissal keys are timezone-independent. Plan step: Docs (AGENTS invariant + READMEs) + TZ-independence regression test.
Fixes review finding #1: an SDManualDay synced from a still-legacy build has no dayKey, so the dayKey-range predicate dropped it before toValue()'s recovery could run — the day vanished from the year report (yet still appeared in a backup) and survived clearYear. manualDays(in:), clear(in:manualDays:), and clearManualDay(_:) now fold in dayKey == nil rows via a shared keylessManualDays helper (recovered with toValue(), filtered in Swift), so such a synced-in day stays visible and deletable. Normally empty; a stopgap for the mixed-version sync window. Adds SwiftDataStore tests for read/clear-year/clear-day recovery, and a Where/TODOs.md entry for the durable successor (per-entity schema versioning with lazy upcasting + minReaderVersion + read-repair).
Fixes review finding #2. manualDays are now keyed by `day` (CalendarDay) instead of a `date` instant, changing the manifest shape, so the format version must bump — otherwise a pre-CalendarDay build reads a new archive as v1 and hits an opaque DecodingError on the missing `date` key instead of the graceful `unsupportedFormatVersion` message. The current reader still imports v1 archives (DayPresence decodes the legacy `date` and recovers its calendar day). Adds a BackupServiceTests case proving a v1 date-keyed manifest imports to the correct CalendarDay (also covers the legacy-import half of finding #3).
Fixes review finding #4. insertLegacyManualDay is purely a test fixture (it writes a deliberately invalid, dayKey-less row), so per the repo convention it belongs in #if DEBUG + @_spi(Testing) and must not ship in release. runMigrations stays ungated — it's the impl runPendingMigrations calls, not test-only.
Addresses review finding #5 (informational). Clarifies in WhereCore/AGENTS.md and DataIssueID.storageKey that CalendarDay pins *stored user records* and their keys across a time-zone change, but GPS samples are still bucketed by the current calendar at read time — so a GPS-only border-drift/abrupt-change dismissal can still shift with its re-bucketed day. Only user-asserted records are travel-proof; bucketing GPS by a fixed zone is intentionally not done.
Fixes review finding #6. The parser validated field widths and 1...12 / 1...31 ranges but accepted well-formed-but-impossible dates like 2026-02-31. It now round-trips the components through the Gregorian calendar and rejects any that normalize (Feb 31, Feb 29 in a non-leap year, Apr 31), while real leap days still parse. Guards against a corrupt persisted key decoding as a bogus day.
Reconsidered for pre-release: the boot-time StoreMigration framework + CalendarDayMigration was over-built for a single developer's data and carried ongoing boot-against-CloudKit risk. It isn't needed for correctness — SDManualDay.toValue() already recovers a legacy dayKey-less row's CalendarDay from dateKey (in UTC, so timezone-stable), which fixes the reappearing-gaps bug on read alone. Persisting canonical dayKeys is handled by a one-time backup export -> replace-import (the export recovers correct days; import rewrites rows with dayKey). Removes: - StoreMigration / CalendarDayMigration and their tests - SwiftDataStore.runPendingMigrations / runMigrations / insertLegacyManualDay + the App-Group version marker, and the WhereBootstrap boot hook - the keyless-row query recovery (manualDays(in:)/clear/clearManualDay) and its tests — a dayKey-less row is invisible to range queries until the one-time reimport rewrites it (allManualDays/export still sees it) Keeps: CalendarDay + threading, toValue() recovery, DataIssueID ISO keys, BackupArchive v2. Docs (AGENTS/README/TODOs) updated to describe the export/reimport approach, note that legacy dismissal keys aren't recovered on read, and keep the per-entity-versioning system as the durable successor.
| /// decides the day boundaries). | ||
| public init(from date: Date, in calendar: Calendar) { | ||
| let parts = calendar.dateComponents([.year, .month, .day], from: date) | ||
| self.init(year: parts.year ?? 1, month: parts.month ?? 1, day: parts.day ?? 1) |
There was a problem hiding this comment.
Should this throw or crash or return nil? Returning 1 for any of these is silent data corruption
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Good catch — changed to a preconditionFailure when the calendar returns no year/month/day (an impossible state for a Gregorian calendar + a concrete Date) rather than silently defaulting to Jan 1. Fixed in 2a84f3a.
| (lhs.year, lhs.month, lhs.day) < (rhs.year, rhs.month, rhs.day) | ||
| } | ||
|
|
||
| public init(from decoder: any Decoder) throws { |
There was a problem hiding this comment.
Do we need custom encoders and decoders? Can't the usual standard compiler provided item do this for us?
There was a problem hiding this comment.
Eg, there's no reason to even round trip to ISO. Just persist the three int fields. If we've initialized, it's a valid day.
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Removed — Codable is now compiler-synthesized over the three int fields (no custom init(from:)/encode(to:)). Fixed in 2a84f3a.
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Done — dropped the ISO round-trip on decode; synthesized Codable persists the three ints directly and trusts validity-by-construction. (init?(iso:) and description stay for the SDManualDay.dayKey / DataIssueID.storageKey string form.) One side effect: the backup manifest's day becomes an object instead of a string — fine since there's no released v2 data. Fixed in 2a84f3a.
| let interval = aggregator.yearInterval(year: year) | ||
| try await store.perform { try await store.clear(in: interval) } | ||
| let dayRange = CalendarDay(year: year, month: 1, day: 1) | ||
| ... CalendarDay(year: year, month: 12, day: 31) |
There was a problem hiding this comment.
Instead of hardcoding this can we derive it from the calendar?
There was a problem hiding this comment.
Posted by an AI agent on kyleve's behalf.
Added CalendarDay.lastDay(ofYear:) / yearRange(_:) and reused them here and in ReportReader, WidgetDataReader, MissingDaysDetector, and MissingDays instead of hardcoding Dec 31 in each spot. Since CalendarDay is Gregorian-only, Dec 31 is invariant, so the helper encodes that once rather than reintroducing a live Calendar dependency for a constant. Fixed in 2a84f3a.
- CalendarDay(from:in:): trap via preconditionFailure when the calendar returns no year/month/day (an impossible state) instead of silently defaulting to Jan 1. - Drop the custom Codable / ISO round-trip; a CalendarDay is valid by construction, so use compiler-synthesized Codable over the three ints. (Persistence/keys still use `description`/`init?(iso:)`; only the backup manifest's `day` shape changes, string -> object — no released v2 data.) - Add CalendarDay.lastDay(ofYear:) / yearRange(_:) helpers and use them in DayJournal.clearYear, ReportReader, WidgetDataReader, MissingDaysDetector, and MissingDays instead of hardcoding Dec 31 in each spot.
## What Trims the largest `AGENTS.md` files down to invariants, hard-fought lessons, and instructions — dropping structure enumerations agents re-derive from the manifests in their first minute of searching — and regroups WhereCore's flat `Sources/` into concern-based subdirectories. One commit per step. **Guiding principle:** an AGENTS.md keeps invariants, lessons, conventions, and recipes; it drops module lists, directory trees, and dependency catalogs that mirror `Package.swift` / `Project.swift`. The root file's libraries list was already stale (missing `WhereIntents` and `SwiftDataInspector`) — proof that content rots. ## Commits 1. **GitHub rules** — new root **GitHub** section: use `gh` for all GitHub interaction; open PRs ready-for-review, never draft; keep open PRs current by pushing each local commit. "Working on PR feedback" now requires replying to a comment when a commit resolves it, and filing deferred feedback durably instead of dropping it. 2. **Root trim** — libraries/targets enumerations become pointers at the manifests; directory layout collapses to the module skeleton. The add-a-target recipe, CI-scheme rule, and the double-linking lesson stay verbatim. 3. **Where trim** — Modules tree becomes a one-paragraph layering stack; developer-overlay tour and feature narration cut; the stylesheet paragraph now points at `WhereUI/AGENTS.md`. 222 → 189 lines. 4. **Module trims** — WhereUI / WhereCore / WhereIntents lose their `Package.swift`-mirroring dependency listings; PeriscopeCore loses its directory catalog. All invariants and the full `WhereStylesheet` guidance stay. 5. **WhereCore regroup** — pure `git mv`, no behavior change: stragglers fold into `Backup/`, `Reminders/`, `Widgets/`, `Location/`, `Evidence/`, `DataResolution/`, `Persistence/`; new `Days/`, `Reporting/`, `Journal/`, `Preferences/` groups. Only `WhereServices*` and `WhereLog` remain top-level. Tests stay flat, matching PeriscopeCore. ## New lessons codified (mined from the last 30 days of PR review) - Prefer synthesized `Codable`; persisted wire formats must survive Swift renames; hand-written conformances document their load-bearing reason (#85, #86) - Retain block-based notification-observer tokens; every `start` has a paired `stop()` (#69) - Group large flat types into sub-structs / child types per behavioral area (#47, #69) - Test doubles conform to the production protocol, never an enum switch in a production type (#54) - Custom full-screen surfaces are accessibility-modal and post focus notifications (#70) - Derive UI dimensions from the live UI (preference keys, `@ScaledMetric`, semantic font styles) instead of repeating magic numbers (#38, #70, #83) - WhereCore: post-write reconciliation is defined once — writes/imports route through `DayJournal.reconcileAfterDayChange()`; cross-collaborator hooks are a single closure (#80, #88) ## Verification - `./swiftformat --lint` clean; `./sync-agents` run after each docs commit - Full `Stuff-iOS-Tests` scheme after the regroup: 1183 tests, 0 failures Made with [Cursor](https://cursor.com)
…store-backed tracked regions (#87) ## What & why Prepares RegionKit for "more locations" ahead of letting users pick their own regions. Two coupled shifts: 1. **Split** the ~2.5 MB monolithic `us-states.geojson` into one GeoJSON file per region, loaded **on demand** — so we only ever parse the regions we track, not the whole US at launch. 2. **Replace** the hardcoded five-case `Region` enum with a **data-driven catalog** (a bundled `regions.json` manifest + a `Region` value type), so any US state (50 + DC + PR) plus Canada/EU are available regions. The user's **tracked** regions live in the synced SwiftData store (WhereCore owns "which"; RegionKit stays the agnostic geometry/lookup engine). Onboarding's region picker and user-chosen per-region styling are **out of scope** here (designed for, not built). ## What changed - **RegionKit — data-driven catalog + on-demand geometry** - `Region` is now a `Hashable`/`Codable` value type over a stable id (`us-CA`, `canada`, …) with a well-known `.other` and conveniences (`.california`, etc.). `RegionCatalog` loads the bundled `regions.json` manifest (all/name/localizedName/geometry file, canonical order). Adding a region is now **pure data** — a manifest + geojson change generated by `Tools/generate-regions.rb`, no code. - `RegionAttributor(for:)` loads only the passed regions' files; `.all` covers the whole catalog, `.shared` the default four. `RegionAttributing` abstracts the engine so the app can supply a live, swappable attributor. `RegionGeometryCatalog.outlines(for:attributor:)` takes an explicit attributor. - **WhereCore — store-backed, synced tracked regions + live attributor** - Tracked regions persist as **one `SDTrackedRegion` row per region** (so concurrent cross-device edits merge instead of last-write-wins), read as a `Set` defaulting to the four until the user chooses. - `RegionAttribution` rebuilds the attributor from the tracked set on the store's `changes()` signal (local edit or remote CloudKit import). `WhereServices.make(...)` (async) derives it from the store; the app launch and the App Intents process (`WhereServices.forIntents()`, now async) both attribute against the same synced set. `make(...)` is the **sole public assembly entry**; the synchronous `init` is `@_spi(Testing)` for tests/previews. - **WhereUI / WhereIntents — catalog-driven** - `RegionStyle` is id-keyed with an isolated, disposable bespoke-override table (the seam user-chosen styling will replace). Ordering uses `Region.inCanonicalOrder(_:)` instead of scanning `Region.allCases`. - Siri's "pick a region" suggestions and the Spotlight index surface the **tracked** set; `RegionEntity` id resolution stays **full-catalog** (so "days in Texas" answers even when untracked). - **Backup** — tracked regions round-trip in the archive (additive field; v1 manifests decode to `[]`); `.replace` restores the set exactly, `.merge` unions it. ## Key decisions / tradeoffs - **Clean id scheme** (`us-CA` vs `california`): existing persisted region raw values change; data is re-exported/re-imported. The `NAME → id` map lives in `Tools/generate-regions.rb`. - **Localization tradeoff:** region names come from the manifest (with an optional `localizationKey` override), so region names lose static string-catalog extraction — consistent with how the App Intents `RegionEntity` already resolves names at runtime. - **Attribution priority** is the manifest's canonical order (regions are mutually exclusive at our resolution). ## Deferred (follow-ups) - The onboarding region picker and user-chosen per-region color/emoji/symbol/tint. - Seeding semantics (the default four apply only at zero rows; the picker will seed/replace) and materializing the implicit default before the first explicit edit. - **Untracking a region:** the store currently *deletes* the tracked-region row, which would re-attribute that region's past GPS days to `.other` on re-aggregation (manual days, stored as region sets, are safe). A `TODO` defers the soft-delete (mark inactive + load every ever-tracked region so history stays stable) to the picker work that defines the untrack UX. Not user-reachable today — nothing calls `setTrackedRegion(false)` outside tests. ## Testing - Full `Stuff-iOS-Tests` scheme green; `./swiftformat --lint` clean. - New/reworked coverage: catalog + value-type (`RegionTests`), subset-only-loads (`RegionAttributorTests`), rebuild-on-`changes()` + store round-trip (`RegionAttributionTests`, `TrackedRegionStoreTests`), end-to-end `make()` wiring (`WhereServicesTests`), tracked-set entity/suggestions (`RegionEntityTests`), and backup round-trip / legacy-manifest decode (`BackupServiceTests`, `BackupCoordinatorTests`). ## Review pass - **Self-review:** log (not swallow) tracked-region read failures; deterministic (canonical-ordered) attributor build; surface unknown stored region ids; backup round-trip (above). - **PR review feedback:** consolidated the RegionKit bundled-data docs into the main `README.md` (removed the separate `Resources/README.md`); documented why `Region`'s `Codable` is hand-written (bare id string vs the synthesized keyed object); made the sync `WhereServices.init` `@_spi(Testing)` so `make()` is the sole public entry; and TODO'd the untrack soft-delete (see Deferred). ## Kept current with `main` Merged `#85` (CalendarDay), `#86` (Periscope JournalKit), `#88` (backup `onImport` reconcile hook), and `#89` (streamlined AGENTS + WhereCore source regroup). Conflicts resolved: the backup `formatVersion` (kept v2 **and** the additive `trackedRegions` field); the `BackupCoordinatorTests` harness swap (kept both the tracked-region tests and `#88`'s `onImport` hook test); and `Where/AGENTS.md` (took `#89`'s streamlined Modules section, but kept the accurate manifest-backed RegionKit localization note over `#89`'s stale "static keys" wording). My changes rode the source regroup into `Backup/`, `Persistence/`, `Days/`, `DataResolution/`, etc.
Why
Manually entered overrides and missing-day backfills (and dismissed issues) were persisted keyed by an absolute
Dateinstant meant to be midnight in the writer's time zone. After a time-zone change (e.g. New York → San Francisco),startOfDayfor a given logical date resolves to a different instant, so stored keys stop matching recomputed day keys and previously-resolved data errors reappear.A DB export confirmed it: of 160 backfilled manual days, 9 were logged in New York (Eastern-midnight instants) and now resolve to the previous calendar day in Pacific — exactly the reappearing gaps.
What
Introduce
CalendarDay(a timezone-independentYYYY-MM-DDvalue) as the identity of a logical day, and key every stored user record and day comparison on it.Dateis kept only where an instant is genuinely needed (GPS sample→day bucketing, calendar-grid geometry, display), always derived viastartOfDay(in:).CalendarDay—Comparable/Hashable/Codable(encodes as the ISO string),Dateconversions, pure-Gregorian arithmetic/ranges, and a legacy start-of-day recovery init.DayPresence.day,RegionDayLocations.day,YearReport,MissingDays/MissingDayRange,PresenceCalendar, the three detectors, andDataIssueID.storageKey(nowprefix:YYYY-MM-DD, so dismissals survive travel).EvidenceReaderday keys too.SDManualDay.dayKey(ISO) is the canonical key;dateKeyis retained as informational history +toValue()'s recovery source. Store CRUD keys manual days byCalendarDay(lexicographic ISO range for year scoping).WhereStore/DayJournal/ReportReader/WidgetDataReaderupdated to match.CalendarDay → Dateonly for display andDatePickerbindings.date→day); a pre-v2 build now refuses a v2 archive with the graceful "newer version" message. The current reader still imports v1 (recovering the legacydate).Existing data (no boot-time migration)
Correctness for pre-
CalendarDayrows comes from read recovery, not a migration:SDManualDay.toValue()recovers a legacydayKey-less row'sCalendarDayfrom itsdateKeyin UTC (timezone-stable), so the reappearing-gaps bug is fixed on read alone. Persisting canonicaldayKeys is a one-time, manual backup export → replace-import (the export recovers correct days; import rewrites rows withdayKey).An earlier revision of this PR added a boot-time
StoreMigrationframework +CalendarDayMigration; it was removed as over-built for pre-release (single developer's data, ongoing boot-against-CloudKit risk) and unnecessary for correctness. The general, durable successor — per-entity schema versioning with lazy upcasting +minReaderVersion+ read-repair — is captured as a follow-up inWhere/TODOs.md.Known gaps (acceptable pre-release, noted in
WhereCore/AGENTS.md+ TODO):dayKey-less row is invisible to thedayKey-range queries until re-imported (allManualDays()/export still see it via recovery), so the year report shows those days after the one-time reimport.CalendarDaydismissal can reappear until re-dismissed or fixed in the export round-trip. (The reported export has 0 dismissals.)Testing
CalendarDayTests,TimeZoneIndependenceTests, plus aBackupServiceTestscase proving a v1date-keyed manifest imports to the correctCalendarDay../swiftformat --lintclean; the fullStuff-iOS-Testsscheme passes on iPhone 17 / iOS 26.2 (the multi-bundle host case).Notes